Skip to content

test: Use sphinx.ext.doctest for doctesting#625

Open
ValerianRey wants to merge 9 commits intomainfrom
rework-doctesting
Open

test: Use sphinx.ext.doctest for doctesting#625
ValerianRey wants to merge 9 commits intomainfrom
rework-doctesting

Conversation

@ValerianRey
Copy link
Contributor

@ValerianRey ValerianRey commented Mar 6, 2026

The idea is to use sphinx.ext.doctest to be able to run examples and check their output.
I think we tried that a very long time ago, but we had two issues:

  • We had some tests that failed due to float numerical errors on some different system. This is less likely to happen now that the value tests of aggregators have moved from doc tests to unit tests (test_values.py). We can also try as much as we can to not have unstable numerical results in the documentation.
  • We couldn't emphasize lines with the doctest directives. Claude has been quite useful this time by extending the directive and adding the emphasize-lines option to it. A more long-term solution would be to contribute this change directly to the sphinx repo, by fixing Add :caption: for .. doctest:: sphinx-doc/sphinx#6915 and testcode directives should support the same options as code-block sphinx-doc/sphinx#6858. But for now, it works.
  • We relied on doc tests to reach 100% code coverage if I remember correctly. This is not the case anymore (we have 100% without doc tests).

In a future step, I'd like to try to also test the examples from the README. I think we could import the entire README to some other place (like the main page of torchjd.org, to avoid having to maintain one). Then, it could be doctested when we run uv run make doctest -C docs. So this would solve two problems at once. But this is out of the scope of this PR, as the README examples were never tested.

Changes:

  • Fix output errors in documentation examples
  • Add sphinx.ext.doctest extension
  • Add emphasize-lines to testcode. This also adds the docutils doc dependency.
  • Remove pytest doc tests
  • Add testcode directives for doctests
  • Update instructions on doc testing
  • Move lightning from test to doc deps
  • Update checks so that the build-doc job runs doctest

TODO before merging: change expected check from "Build documentation" to "Build and test documentation" in ruleset.

@ValerianRey ValerianRey added the cc: test Conventional commit type for changes to tests. label Mar 6, 2026
@ValerianRey ValerianRey self-assigned this Mar 6, 2026
@ValerianRey ValerianRey added the cc: test Conventional commit type for changes to tests. label Mar 6, 2026
@github-actions github-actions bot changed the title Use sphinx.ext.doctest for doctesting test: Use sphinx.ext.doctest for doctesting Mar 6, 2026
Import several classes from ``torch`` and ``torchjd``:

.. code-block:: python
.. testcode::
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this change the way it looks?

Copy link
Contributor Author

@ValerianRey ValerianRey Mar 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this change the way it looks?

I don't remember. It looks good at least, and doesn't change much. I'll compare later today or tomorrow.

Copy link
Contributor

@PierreQuinton PierreQuinton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice, didn't check the way code looks in the doc, but otherwise LGTM

@ValerianRey
Copy link
Contributor Author

Something I forgot to mention is that by default (i.e. "default"), the python code that is after >>> is also tested. So some places have become tested even though there is no diff for them in the PR.

We could also disable this + use .. doctest:: directives for these examples, so that we're able to decide which of them to test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cc: test Conventional commit type for changes to tests.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve doctesting

2 participants